Methods and systems for measuring brand uplift for search campaigns

ABSTRACT

Systems and methods for measuring brand uplift in a computer networked environment are provided. A data processing system can assign a first set of device identifiers to an experimental arm and a second set of device identifiers to a control arm. Computing devices associated with the experimental arm receive third-party content items promoting a brand for which a brand uplift experiment is performed, while computing devices associated with the control arm do not receive third-party content items promoting the brand. At the end of the experiment, a brand uplift metric value is computed by analyzing search query logs of the computing devices. The brand uplift metric value is then provided to an advertiser for display.

CROSS-REFERENCE TO RELATED PATENT APPLICATION

The present application is a continuation of and claims priority to U.S.patent application Ser. No. 14/485,164, entitled “METHODS AND SYSTEMSFOR MEASURING BRAND UPLIFT FOR SEARCH CAMPAIGNS”, filed Sep. 12, 2014,which is incorporated herein by reference in its entirety.

BACKGROUND

In a computer networked environment such as the internet, third-partycontent providers provide third-party content items for display onend-user computing devices. These third-party content items, forexample, advertisements, can be linked to web pages associated with thethird-party content providers. These third-party content items caninclude content identifying the third-party content provider thatprovided the content item.

SUMMARY

At least one aspect is directed to a computer implemented method formeasuring brand uplift in a computer network environment. According toone aspect, a data processing system including one or more processorscan receive a plurality of requests for content from a plurality ofcomputing devices associated with device identifiers. The dataprocessing system can identify, for each of the plurality of requestsfor content, a third-party content item of a first set of content itemssuitable for display with the content associated with the request. Eachof the first set of content items identifies a first brand. The dataprocessing system can determine that a first brand promoted by theidentified third-party content item is participating in a brand upliftexperiment. The data processing system can determine, responsive toreceiving the requests for content, to display third-party content itemsof the first set of content items to a first subset of the plurality ofcomputing devices, the first subset of the plurality of computingdevices associated with a first subset of the device identifiers. Thedata processing system can provide, for display, the third-party contentitems of the first set of content items at the first subset of theplurality of computing devices responsive to determining to display thethird-party content items of the first set of content items to the firstsubset of the plurality of computing devices. The data processing systemcan determine to not display the first set of content items to aremaining subset of the plurality of computing devices that areassociated with a remaining subset of the device identifiers. The dataprocessing system can select, for the remaining subset of the pluralityof computing devices, third-party content items of a second set ofcontent items, none of the second set of content items identifying thefirst brand. The data processing system can identify, from a searchquery log including search queries associated with the deviceidentifiers corresponding to the plurality of computing devices, aportion of the first subset of device identifiers that performed asearch using a search query that included a keyword from a predeterminedset of brand keywords associated with the first brand. The dataprocessing system can determine a first brand awareness metric valuebased on a ratio of a number of device identifiers in the portion of thefirst subset of device identifiers to a total number of deviceidentifiers in the first subset of device identifiers. The dataprocessing system can identify, from the search query log, a portion ofthe second subset of device identifiers that performed a search using asearch query that included a keyword from the predetermined set of brandkeywords associated with the first brand. The data processing system candetermine a second brand awareness metric value based on a ratio of anumber of device identifiers in the portion of the second subset ofdevice identifiers to a total number of device identifiers in the secondsubset of device identifiers. The data processing system can compute abrand uplift metric value indicating an uplift in brand awareness of thefirst brand identified by the first set of content items based on thefirst brand awareness metric value and the second brand awareness metricvalue. The data processing system can display, to a content provider ofthe first set of content items, data based on the brand uplift metricvalue indicating the uplift in brand awareness of the first brand.

In some implementations, the search query log includes search queriesassociated with a device identifier that were submitted with a searchrequest after the third-party content item of the first set of contentitems was displayed at a computing device associated with the deviceidentifier.

In some implementations, determining the first brand awareness metricvalue includes providing, to the first subset of device identifiers, asurvey including information associated with the first brand, receiving,from the first subset of device identifiers, survey responses anddetermining, from the survey responses, a ratio of a number of deviceidentifiers in a portion of the first subset of device identifiers thatidentified the brand to the total number of device identifiers in thefirst subset of device identifiers, the first brand awareness metricvalue based on the determined ratio.

In some implementations, determining the second brand awareness metricvalue includes providing, to the second subset of device identifiers, asurvey including information associated with the first brand, receiving,from the second subset of device identifiers, survey responses anddetermining, from the survey responses, a ratio of a number of deviceidentifiers in a portion of the second subset of device identifiers thatidentified the first brand to the total number of device identifiers inthe second subset of device identifiers, the second brand awarenessmetric value based on the determined ratio.

In some implementations, the data processing system can maintain, forthe first subset of device identifiers, a counter identifying a numberof computing devices associated with the first subset of deviceidentifiers on which any of the third-party content items of the firstset of content items was displayed and increment the counter when acontent item receives a valid impression on a computing device on whichnone of the first set of content items has previously received a validimpression. In some such implementations, incrementing the counter whena content item receives a valid impression on a computing device onwhich none of the first set of content items has previously received avalid impression includes determining, responsive to displaying one ofthe third-party content items of the first set of content items at thefirst subset of the plurality of computing devices, a length of timefrom a time a third-party content items of the first set of contentitems was displayed on an information resource and a request to accessanother content item included in the information resource is receivedand determining that the length of time is above a threshold length oftime.

In some implementations, the data processing system can generate thepredetermined set of brand keywords associated with the first brand byanalyzing content associated with the first brand.

At least one aspect is directed to a system of measuring brand uplift ina computer network environment. The system includes a data processingsystem including a content selection module and a brand upliftcalculation module. The data processing system can identify, for each ofthe plurality of requests for content, a third-party content item of afirst set of content items suitable for display with the contentassociated with the request. Each of the first set of content itemsidentifies a first brand. The data processing system can determine thata first brand promoted by the identified third-party content item isparticipating in a brand uplift experiment. The data processing systemcan determine, responsive to receiving the requests for content, todisplay third-party content items of the first set of content items to afirst subset of the plurality of computing devices, the first subset ofthe plurality of computing devices associated with a first subset of thedevice identifiers. The data processing system can provide, for display,the third-party content items of the first set of content items at thefirst subset of the plurality of computing devices responsive todetermining to display the third-party content items of the first set ofcontent items to the first subset of the plurality of computing devices.The data processing system can determine to not display the first set ofcontent items to a remaining subset of the plurality of computingdevices that are associated with a remaining subset of the deviceidentifiers. The data processing system can select, for the remainingsubset of the plurality of computing devices, third-party content itemsof a second set of content items, none of the second set of contentitems identifying the first brand. The data processing system canidentify, from a search query log including search queries associatedwith the device identifiers corresponding to the plurality of computingdevices, a portion of the first subset of device identifiers thatperformed a search using a search query that included a keyword from apredetermined set of brand keywords associated with the first brand. Thedata processing system can determine a first brand awareness metricvalue based on a ratio of a number of device identifiers in the portionof the first subset of device identifiers to a number of total deviceidentifiers in the first subset of device identifiers. The dataprocessing system can identify, from the search query log, a portion ofthe second subset of device identifiers that performed a search using asearch query that included a keyword from the predetermined set of brandkeywords associated with the first brand. The data processing system candetermine a second brand awareness metric value based on a ratio of anumber of device identifiers in the portion of the second subset ofdevice identifiers to a number of total device identifiers in the secondsubset of device identifiers. The data processing system can compute abrand uplift metric value indicating an uplift in brand awareness of thefirst brand identified by the first set of content items based on thefirst brand awareness metric value and the second brand awareness metricvalue. The data processing system can display, to a content provider ofthe first set of content items, data based on the brand uplift metricvalue indicating the uplift in brand awareness of the first brand.

At least one aspect is directed to a computer readable storage mediumstoring instructions that when executed by one or more data processors,cause the one or more data processors to perform operations. The dataprocessing system can identify, for each of the plurality of requestsfor content, a third-party content item of a first set of content itemssuitable for display with the content associated with the request. Eachof the first set of content items identifies a first brand. The dataprocessing system can determine that a first brand promoted by theidentified third-party content item is participating in a brand upliftexperiment. The data processing system can determine, responsive toreceiving the requests for content, to display third-party content itemsof the first set of content items to a first subset of the plurality ofcomputing devices, the first subset of the plurality of computingdevices associated with a first subset of the device identifiers. Thedata processing system can provide, for display, the third-party contentitems of the first set of content items at the first subset of theplurality of computing devices responsive to determining to display thethird-party content items of the first set of content items to the firstsubset of the plurality of computing devices. The data processing systemcan determine to not display the first set of content items to aremaining subset of the plurality of computing devices that areassociated with a remaining subset of the device identifiers. The dataprocessing system can select, for the remaining subset of the pluralityof computing devices, third-party content items of a second set ofcontent items, none of the second set of content items identifying thefirst brand. The data processing system can identify, from a searchquery log including search queries associated with the deviceidentifiers corresponding to the plurality of computing devices, aportion of the first subset of device identifiers that performed asearch using a search query that included a keyword from a predeterminedset of brand keywords associated with the first brand. The dataprocessing system can determine a first brand awareness metric valuebased on a ratio of a number of device identifiers in the portion of thefirst subset of device identifiers to a number of total deviceidentifiers in the first subset of device identifiers. The dataprocessing system can identify, from the search query log, a portion ofthe second subset of device identifiers that performed a search using asearch query that included a keyword from the predetermined set of brandkeywords associated with the first brand. The data processing system candetermine a second brand awareness metric value based on a ratio of anumber of device identifiers in the portion of the second subset ofdevice identifiers to a number of total device identifiers in the secondsubset of device identifiers. The data processing system can compute abrand uplift metric value indicating an uplift in brand awareness of thefirst brand identified by the first set of content items based on thefirst brand awareness metric value and the second brand awareness metricvalue. The data processing system can display, to a content provider ofthe first set of content items, data based on the brand uplift metricvalue indicating the uplift in brand awareness of the first brand.

These and other aspects and implementations are discussed in detailbelow. The foregoing information and the following detailed descriptioninclude illustrative examples of various aspects and implementations,and provide an overview or framework for understanding the nature andcharacter of the claimed aspects and implementations. The drawingsprovide illustration and a further understanding of the various aspectsand implementations, and are incorporated in and constitute a part ofthis specification.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are not intended to be drawn to scale. Likereference numbers and designations in the various drawings indicate likeelements. For purposes of clarity, not every component may be labeled inevery drawing. In the drawings:

FIG. 1 is a block diagram depicting one implementation of an environmentfor measuring brand uplift for search campaigns, according to anillustrative implementation;

FIG. 2A is a display depicting an online search result documentincluding a third-party content item identifying a first brand,according to an illustrative implementation;

FIG. 2B is a display depicting an online search result documentincluding a third-party content item not identifying the first brand,according to an illustrative implementation;

FIGS. 3A and 3B are flow diagrams depicting a method of measuring branduplift for search campaigns, according to an illustrativeimplementation; and

FIG. 4 is a block diagram illustrating a general architecture for acomputer system that may be employed to implement elements of thesystems and methods described and illustrated herein, according to anillustrative implementation.

DETAILED DESCRIPTION

Following below are more detailed descriptions of various conceptsrelated to, and implementations of, methods, apparatuses, and systems ofmeasuring brand uplift for search campaigns in a computer networkenvironment. The various concepts introduced above and discussed ingreater detail below may be implemented in any of numerous ways, as thedescribed concepts are not limited to any particular manner ofimplementation.

Third-party content providers, such as advertisers, may focus theiradvertising campaigns on various goals, including but not limited toperformance advertising and brand advertising. Traditionally,advertisers whose ads are displayed in response to search queries arefocused on performance advertising, in which the goal of the ads isprimarily to direct a user to a landing page of the ad. However, adsthat are displayed on search result web pages can also indirectly createbrand advertising opportunities for the advertiser. One purpose of brandadvertising is to increase brand awareness to consumers. As such, withbrand advertising, advertisers may not necessarily care about a clickthrough rate as the goal is not necessarily to convert a consumer into acustomer, but simply to increase the awareness of the brand associatedwith the advertiser. For instance, a user may submit a search query“transfer money to India.” In response to the search query, ads fromvarious brands may be displayed along with search results generated bythe search engine. Although the user may not click on any of the adsdisplayed on the search results page, the user may view the ads and thebrands identified in the ads such that the user may recall one or moreof the brands when performing a future search relating to transferringmoney to India.

At present, there are not enough measurement metrics to quantify theimpact that search ads contribute towards brand advertising. The presentdisclosure aims to address this by providing experimental methodologiesto find effective metrics to measure brand metrics like brand recall andbrand awareness for search advertising.

Systems and methods of the present disclosure relate generally tomeasuring brand uplift for search campaigns in a computer networkenvironment. A data processing system including one or more processorscan receive a plurality of requests for content from a plurality ofcomputing devices associated with device identifiers. The dataprocessing system can identify, for each of the plurality of requestsfor content, a third-party content item of a first set of content itemssuitable for display with the content associated with the request. Eachof the first set of content items identifies a first brand. The dataprocessing system can determine that a first brand promoted by theidentified third-party content item is participating in a brand upliftexperiment. The data processing system can determine, responsive toreceiving the requests for content, to display third-party content itemsof the first set of content items to a first subset of the plurality ofcomputing devices, the first subset of the plurality of computingdevices associated with a first subset of the device identifiers. Thedata processing system can provide, for display, the third-party contentitems of the first set of content items at the first subset of theplurality of computing devices responsive to determining to display thethird-party content items of the first set of content items to the firstsubset of the plurality of computing devices. The data processing systemcan determine to not display the first set of content items to aremaining subset of the plurality of computing devices that areassociated with a remaining subset of the device identifiers. The dataprocessing system can select, for the remaining subset of the pluralityof computing devices, third-party content items of a second set ofcontent items, none of the second set of content items identifying thefirst brand. The data processing system can identify, from a searchquery log including search queries associated with the deviceidentifiers corresponding to the plurality of computing devices, aportion of the first subset of device identifiers that performed asearch using a search query that included a keyword from a predeterminedset of brand keywords associated with the first brand. The dataprocessing system can determine a first brand awareness metric valuebased on a ratio of the portion of the first subset of deviceidentifiers to the first subset of device identifiers. The dataprocessing system can identify, from the search query log, a portion ofthe second subset of device identifiers that performed a search using asearch query that included a keyword from the predetermined set of brandkeywords associated with the first brand. The data processing system candetermine a second brand awareness metric value based on a ratio of theportion of the second subset of device identifiers to the second subsetof device identifiers. The data processing system can compute a branduplift metric value indicating an uplift in brand awareness of the firstbrand identified by the first set of content items based on the firstbrand awareness metric value and the second brand awareness metricvalue. The data processing system can display, to a content provider ofthe first set of content items, the brand uplift metric value indicatingthe uplift in brand awareness of the first brand.

FIG. 1 is a block diagram depicting one implementation of an environmentfor measuring brand uplift for search campaigns. The environment 100includes at least one data processing system 110. The data processingsystem 110 can include at least one processor and a memory, i.e., aprocessing circuit. The memory stores processor-executable instructionsthat, when executed by processor, cause the processor to perform one ormore of the operations described herein. The processor may include amicroprocessor, application-specific integrated circuit (ASIC),field-programmable gate array (FPGA), etc., or combinations thereof. Thememory may include, but is not limited to, electronic, optical,magnetic, or any other storage or transmission device capable ofproviding the processor with program instructions. The memory mayfurther include a floppy disk, CD-ROM, DVD, magnetic disk, memory chip,ASIC, FPGA, read-only memory (ROM), random-access memory (RAM),electrically-erasable ROM (EEPROM), erasable-programmable ROM (EPROM),flash memory, optical media, or any other suitable memory from which theprocessor can read instructions. The instructions may include code fromany suitable computer-programming language. The data processing systemcan include one or more computing devices or servers that can performvarious functions. In some implementations, the data processing systemcan include an advertising auction system configured to host auctions.In some implementations, the data processing system does not include theadvertising auction system but is configured to communicate with theadvertising auction system via the network 105.

The network 105 can include computer networks such as the internet,local, wide, metro or other area networks, intranets, satellitenetworks, other computer networks such as voice or data mobile phonecommunication networks, and combinations thereof. The data processingsystem 110 of the system 100 can communicate via the network 105, forinstance with at least one content provider computing device 115, atleast one content publisher computing device 120, or at least one enduser computing device 125. The network 105 may be any form of computernetwork that relays information between the user computing device 115,data processing system 110, and one or more content sources, forexample, web servers, advertising servers, amongst others. For example,the network 105 may include the Internet and/or other types of datanetworks, such as a local area network (LAN), a wide area network (WAN),a cellular network, satellite network, or other types of data networks.The network 105 may also include any number of computing devices (e.g.,computer, servers, routers, network switches, etc.) that are configuredto receive and/or transmit data within network 105. The network 105 mayfurther include any number of hardwired and/or wireless connections. Forexample, the user computing device 115 may communicate wirelessly (e.g.,via WiFi, cellular, radio, etc.) with a transceiver that is hardwired(e.g., via a fiber optic cable, a CATS cable, etc.) to other computingdevices in network 105.

The content provider computing devices 115 can include servers or othercomputing devices operated by a content provider entity to providecontent items such as advertisements for display on informationresources at the end user computing device 125. The content provided bythe content provider computing device 115 can include third partycontent items or creatives (e.g., ads) for display on informationresources such as a website or web page that includes primary content,e.g. content provided by the content publisher computing device 120. Thecontent items can also be displayed on a search results web page. Forinstance, the content provider computing device 115 can provide or bethe source of ads or other content items for display in content slots ofcontent web pages such as a web page of a company where the primarycontent of the web page is provided by the company, or for display on asearch results landing page provided by a search engine. The contentitems associated with the content provider computing device 115 can bedisplayed on information resources other than web pages, such as contentdisplayed as part of the execution of an application on a smartphone orother end user computing device 125.

The content publisher computing devices 120 can include servers or othercomputing devices operated by a content publishing entity to provideprimary content for display via the network 105. For instance, thecontent publisher computing device 120 can include a web page operatorwho provides primary content for display on the web page. The primarycontent can include content other than that provided by the contentpublisher computing device 120, and the web page can include contentslots configured for the display of third party content items (e.g.,ads) from the content provider computing devices 115. For instance, thecontent publisher computing device 120 can operate the website of acompany and can provide content about that company for display on webpages of the website. The web pages can include content slots configuredfor the display of third party content items such as ads of the contentprovider computing device 115. In some implementations, the contentpublisher computing device 120 includes a search engine computing device(e.g. server) of a search engine operator that operates a search enginewebsite. The primary content of search engine web pages (e.g., a resultsor landing web page) can include results of a search as well as thirdparty content items displayed in content slots such as content itemsfrom the content provider computing device 115.

The end user computing devices 125 can include computing devicesconfigured to communicate via the network 105 to display data such asthe content provided by the content publisher computing device 120(e.g., primary web page content or other information resources) and thecontent provided by the content provider computing device 115 (e.g.,third party content items such as ads configured for display in acontent slot of a web page). The end user computing device 125, thecontent provider computing device 115, and the content publishercomputing device 120 can include desktop computers, laptop computers,tablet computers, smartphones, personal digital assistants, mobiledevices, consumer computing devices, servers, clients, digital videorecorders, a set-top box for a television, a video game console, or anyother computing device configured to communicate via the network 105.

The content provider computing devices 115, the content publishercomputing device 120 and the end user computing devices 125 can includea processor and a memory, i.e., a processing circuit. The memory storesmachine instructions that, when executed by processor, cause processorto perform one or more of the operations described herein. The processormay include a microprocessor, application-specific integrated circuit(ASIC), field-programmable gate array (FPGA), etc., or combinationsthereof. The memory may include, but is not limited to, electronic,optical, magnetic, or any other storage or transmission device capableof providing the processor with program instructions. The memory mayfurther include a floppy disk, CD-ROM, DVD, magnetic disk, memory chip,ASIC, FPGA, read-only memory (ROM), random-access memory (RAM),electrically-erasable ROM (EEPROM), erasable-programmable ROM (EPROM),flash memory, optical media, or any other suitable memory from which theprocessor can read instructions. The instructions may include code fromany suitable computer-programming language.

The content provider computing devices 115, the content publishercomputing devices 120 and the end user computing devices 125 may alsoinclude one or more user interface devices. In general, a user interfacedevice refers to any electronic device that conveys data to a user bygenerating sensory information (e.g., a visualization on a display, oneor more sounds, etc.) and/or converts received sensory information froma user into electronic signals (e.g., a keyboard, a mouse, a pointingdevice, a touch screen display, a microphone, etc.). The one or moreuser interface devices may be internal to a housing of the contentprovider computing devices 115, the content publisher computing device120 and the end user computing devices 125 (e.g., a built-in display,microphone, etc.) or external to the housing of content providercomputing devices 115, the content publisher computing device 120 andthe end user computing devices 125 (e.g., a monitor connected to theuser computing device 115, a speaker connected to the user computingdevice 115, etc.), according to various implementations. For example,the content provider computing devices 115, the content publishercomputing device 120 and the end user computing devices 125 may includean electronic display, which visually displays web pages using webpagedata received from one or more content sources and/or from the dataprocessing system 110 via the network 105. In some implementations, acontent placement campaign manager or third-party content provider, suchas an advertiser, can communicate with the data processing system 110via the content provider computing devices 115. In some implementations,the advertiser can communicate with the data processing system 110 via auser interface displayed on the user interface devices of the contentprovider computing devices 115.

The data processing system 110 can include at least one server. Forinstance, the data processing system 110 can include a plurality ofservers located in at least one data center or server farm. In someimplementations, the data processing system 110 includes a contentplacement system, e.g., an ad server or ad placement system. The dataprocessing system 110 can include at least one content selection module130, at least one brand uplift calculation module 135, and at least onedatabase 140. The content selection module 130 and brand upliftcalculation module 135 can each include at least one processing unit,server, virtual server, circuit, engine, agent, appliance, or otherlogic device such as programmable logic arrays configured to communicatewith the database 140 and with other computing devices (e.g., thecontent provider computing device 115, the content publisher computingdevice 120, or the end user computing device 125) via the network 105.

The content selection module 130 and brand uplift calculation module 135can include or execute at least one computer program or at least onescript. The content selection module 130 and brand uplift calculationmodule 135 can be separate components, a single component, or part ofthe data processing system 110. The content selection module 130 andbrand uplift calculation module 135 can include combinations of softwareand hardware, such as one or more processors configured to execute oneor more scripts to receive a plurality of requests for content from aplurality of computing devices associated with device identifiers,identify, for each of the plurality of requests for content, athird-party content item of a first set of content items suitable fordisplay with the content associated with the request, determine that afirst brand promoted by the identified third-party content item isparticipating in a brand uplift experiment, determine, responsive toreceiving the requests for content, to display third-party content itemsof the first set of content items to a first subset of the plurality ofcomputing devices, the first subset of the plurality of computingdevices associated with a first subset of the device identifiers,provide, for display, the third-party content items of the first set ofcontent items at the first subset of the plurality of computing devicesresponsive to determining to display the third-party content items ofthe first set of content items to the first subset of the plurality ofcomputing devices, determine to not display the first set of contentitems to a remaining subset of the plurality of computing devices thatare associated with a remaining subset of the device identifiers,select, for the remaining subset of the plurality of computing devices,third-party content items of a second set of content items, none of thesecond set of content items identifying the first brand, identify, froma search query log including search queries associated with the deviceidentifiers corresponding to the plurality of computing devices, aportion of the first subset of device identifiers that performed asearch using a search query that included a keyword from a predeterminedset of brand keywords associated with the first brand, determine a firstbrand awareness metric value based on a ratio of the portion of thefirst subset of device identifiers to the first subset of deviceidentifiers, identify, from the search query log, a portion of thesecond subset of device identifiers that performed a search using asearch query that included a keyword from the predetermined set of brandkeywords associated with the first brand, determine a second brandawareness metric value based on a ratio of the portion of the secondsubset of device identifiers to the second subset of device identifiers,compute a brand uplift metric value indicating an uplift in brandawareness of the first brand identified by the first set of contentitems based on the first brand awareness metric value and the secondbrand awareness metric value, display, to a content provider of thefirst set of content items, the brand uplift metric value indicating theuplift in brand awareness of the first brand, for example.

The data processing system can also include one or more contentrepositories or databases 140. The databases 140 can be local to thedata processing system 110. In some implementations, the databases 140can be remote to the data processing system 110 but can communicate withthe data processing system 110 via the network 105. The databases 140can store reservations for a plurality of content slots available forreservation on a plurality of information resources as well as storeavailability schedules for the plurality of content slots. Additionaldetails of the contents of the databases 140 will be provided below.

The data processing system 110 can be configured to establish a branduplift experiment for one or more third-party content providers 115 oradvertisers. In some implementations, the data processing system 110 canreceive a request from an advertiser to establish a brand upliftexperiment. The data processing system 110 can receive a list of brandkeywords identifying one or more keywords that the advertiser eitherassociates or is likely to associate with the brand. The data processingsystem 110 can receive the list from the advertiser. In someimplementations, the data processing system 110 can generate the list ofbrand keywords by analyzing the advertiser's web pages. In someimplementations, the data processing system 110 can also generate thelist of brand keywords by analyzing the advertiser's third-party contentitems that the advertiser has provided for selection to display at enduser computing devices 125. Generally, the brand keywords can includekeywords that are trademarks of the advertiser, keywords or phrasescommonly attributed to the brand, or other keywords or phrases that theadvertiser believes uniquely identifies the brand from other competingbrands.

In addition, the advertiser may select one or more parameters for thebrand uplift experiment. In some implementations, the advertiser canlimit the scope of the brand uplift experiment to include end usercomputing devices or users based on geography, time of day, and type ofdevice, among others. In some implementations, the advertiser may alsolimit the scope of the experiment to end user computing devices or usershaving certain characteristics. In some implementations, the dataprocessing system 110 can establish the parameters of the brand upliftexperiment without receiving input from the advertiser. In someimplementations, the brand uplift experiment can be set up so that onlya predetermined number of content placement opportunities are usedtowards the experiment to increase the number of impressions third-partycontent items of the advertiser may receive.

To the extent that the systems discussed here receive or collectpersonal information about visitors, or may make use of personalinformation, the visitors may be provided with an opportunity to controlwhether programs or features that may collect personal information(e.g., information about a user's social network, social actions oractivities, a user's preferences, or a user's current location), or tocontrol whether and/or how to receive content from the content serverthat may be more relevant to the user. In addition, certain data may beanonymized in one or more ways before it is stored or used, so thatpersonally identifiable information is removed when generatingparameters (e.g., demographic parameters). For example, a user'sidentity may be anonymized so that no personally identifiableinformation can be determined for the user, or a user's geographiclocation may be generalized where location information is obtained (suchas to a city, ZIP code, or state level), so that a particular locationof a user cannot be determined. Thus, the user may have control over howinformation is collected about him or her and used by a content server.

In establishing the brand uplift experiment for a given advertiser, thedata processing system 110 can identify third-party content items of theadvertiser that are participating in the experiment. These content itemsmay be selected for display but eventually not displayed by virtue ofbeing involved in the experiment. In some implementations, thethird-party content items can be designed or configured to includecontent, such as keywords, that identify the advertiser. In someimplementations, the third-party content items can be designed orconfigured to include content that includes one or more brand keywords.

Once the brand uplift experiment is live, the data processing system 110can begin running the experiment. In general operation of the dataprocessing system 110, the data processing system 110 is receivingrequests to provide content to a plurality of end user computing devicesassociated with unique device identifiers. The data processing system110 can receive the request and identify a search query included in therequest. Using the search query, the data processing system 110 canidentify third-party content items that are deemed relevant to displayat the end user computing device associated with the search query. Insome implementations, a third-party content item can be deemed relevantto a search query if the advertiser of the third-party content itemplaced a bid to serve the third-party content item with one or more ofthe keywords in the search query. For instance, if the search query is“money transfer to India,” the advertiser can identify a plurality ofthird-party content items that have bid on one or more keywords “moneytransfer,” “transfer money,” “money transfer to India” and “transfermoney to India.” In typical operation, the data processing system 110may be configured to select the third-party content items having thehighest scores, which can be based on the bid of the content item andthe relevancy of the content item to the search query. In someimplementations, the data processing system 110 can select apredetermined number of content items to display with the search resultscorresponding to the search query. The content items may be positionedon the display based on their scores.

Referring briefly to FIG. 2A, a display depicting an online searchresult document including a third-party content item identifying a firstbrand is shown. The display 200 includes a search results informationresource 202. The search results information resource includes a searchquery input region 204, a search results content region 206 in whichsearch results are displayed and one or more third-party content regions210 and 220 that are configured to include one or more third-partycontent items. The content region 210 can be positioned above the searchresults content region 206 and can include one or more third-partycontent slots, such as the content slot 213. The third-party contentregion 210 includes a first third-party content item 212 correspondingto an ad provided by a first advertiser, “Advertiser1,” a secondthird-party content item 214 corresponding to an ad provided by a secondadvertiser, “Advertiser2,” that is provided in the content slot 213, anda third third-party content item 216 corresponding to an ad provided bya third advertiser, “Advertiser3.” Moreover, the search resultsinformation resource 202 also includes another third-party contentregion 220 that includes a plurality of third-party content itemsprovided by the three advertisers that provided content items in thethird-party content region 210. The data processing system 110 can beconfigured to select the third-party content items displayed in theregions 210 and 220.

Now, referring again also to FIG. 1, the content selection module 130 ofthe data processing system 110 can be configured to identify athird-party content item that corresponds to a brand participating in abrand uplift experiment and determine whether to provide the third-partycontent item for display at an end user computing device associated withthe search request by which the data processing system 110 identifiedthe third-party content item. The content selection module 130 candetermine whether or not to provide the content item for display basedon whether a device identifier of the end user computing device isassigned to an experimental arm or a control arm of the brand upliftexperiment corresponding to the brand which the content item ispromoting.

The content selection module 130 can be configured to receive aplurality of requests for content from a plurality of computing devicesassociated with device identifiers. Each request corresponds to arequest to provide third-party content items for display at an end usercomputing device. Each request can identify a search query based onwhich the content selection module 130 is configured to select contentitems. The request also identifies a device identifier, which thecontent selection module 130 can use to determine whether or not toprovide one or more of the content items selected by the contentselection module.

The content selection module 130 can receive a request for content inresponse to a request from an end user computing device. In a typicalflow of events, an end user, via the end user computing device 125, cansubmit a search query in a search query information resource, such as aweb page that includes a search query input region in which a searchquery can be inserted. In some implementations, the search query inputregion can be a part of a web browser executing on the end usercomputing device 125. The end user computing device 125 can transmit thesearch query to a content publisher, such as the content publisher 120associated with the search query input region. The content publisher 120can then generate search results based on the search query and generatea search results information resource that includes the search results.The content publisher 120 can provide the search results informationresource to the end user computing device 125, which then displays thecontents of the search results information resource on a display of theend user computing device 125. The content publisher 120 can configurethe search results information resource to include one or more contentslots that are configured to display third-party content items withinthe content slots. These content slots can be located at variouslocations within the search results information resource.

The request for content can include a request to provide third-partycontent alongside search results that are to be displayed in a searchresults information resource. In some implementations, the requests forcontent can be received directly from end user computing devicesassociated with device identifiers. In some implementations, therequests for content can be received via another entity, such as thecontent publisher responsive to the content publisher receiving arequest for search results based on a search query. In someimplementations, the end user computing device 125 can transmit therequest for search results based on the search query to the contentpublisher 120 via the network 105. In turn, the content publisher 120can transmit a request to the data processing system 110 for third-partycontent items to provide in the search results information resourceincluding the search results generated by the content publisherresponsive to the search query. The content selection module 130 canthen select one or more third-party content items provided bythird-party content providers 115 for display on the search resultsinformation resource. Details of the selection process are providedherein.

In some implementations, the request for search results generated by theend user computing device 125 can include the search query. In addition,the request for search results can include a device identifieridentifying the end user computing device 125. In some implementations,the device identifier can be unique to the end user computing device. Insome implementations, the device identifier can be unique to a user ofthe end user computing device. In some implementations, the deviceidentifier can be unique to a particular application, for example, a webbrowser of the end user computing device. The request for third-partycontent received by the data processing system 110 can include thedevice identifier associated with the end user computing device 125 thatrequested search results. In some implementations, the end usercomputing device 125 can transmit the request for third-party content tothe data processing system 110. In some implementations, the contentpublisher 120 receiving the request for search results from the end usercomputing device 125 can transmit the request for third-party content tothe data processing system 110.

Responsive to receiving a given request of the plurality of requests forthird-party content, the content selection module 130 can identify thesearch query included in the request and the device identifieridentified in the request. The content selection module 130 can thenidentify, via an advertising auction system, one or more third-partycontent items that are suitable for display with the search results. Insome implementations, the content selection module 130 can identifythird-party content items via an auction. In some implementations, anauction can be based on the search query, such that advertisers canprovide bids to serve third-party content items on the same informationresource as search results to the search query provided by the contentpublisher. The highest bidding content items may then be selected fordisplay.

In some implementations, the content selection module 130 can identify,for the given request, at least one third-party content item of a firstset of content items suitable for display on the information resourceincluding the search results to the search query. Each of the first setof content items can identify a brand participating in a brand upliftexperiment. For a given third-party content item of the third-partycontent items of the first set of content items identified by thecontent selection module 130 as suitable for display with search resultscorresponding to the search query, the content selection module 130 candetermine the brand the third-party content item is promoting. Thecontent selection module 130 can do so by analyzing the content of thethird-party content item or by analyzing the web address or contents ofthe landing page to which the third-party content item is linked. Insome implementations, the content selection module 130 can determine thebrand or advertiser the third-party content item is promoting byidentifying an advertisement campaign in which the third-party contentitem is included.

The content selection module 130 can be configured to determine that thebrand corresponding to the third-party content item is participating ina brand uplift experiment. Once the content selection module 130determines the identity of the brand or advertiser associated with thecontent item as described above, the content selection module 130 candetermine if the brand is participating in a brand uplift experiment. Insome implementations, the content selection module 130 can perform alookup in a database or table that includes a list of brands that areparticipating in a brand uplift experiment. If the brand is included inthe list of brands, the content selection module 130 can determine thatthe brand is participating in a brand uplift experiment.

The content selection module 130 can be configured to identify, for thegiven request, one or more of the identified third-party content itemsthat are associated with a brand that is participating in a brand upliftexperiment. To do so, the content selection module 130 can identify, foreach of the identified third-party content items, the brand associatedwith the third-party content item. If the third-party content item isnot associated with a brand on a list of brands participating in a branduplift experiment, the content selection module 130 can determine thatthe third-party content item can be provided for display. Conversely, ifthe third-party content item is associated with a brand on the list ofbrands participating in a brand uplift experiment, the content selectionmodule 130 has to execute a set of instructions corresponding to thebrand uplift experiment. In some implementations, the content selectionmodule 130 can maintain a list of brands that are participating in abrand uplift experiment. It should be appreciated that the third-partycontent items can include content that identifies a particular brand. Insome implementations, the third-party content item can include text thatidentifies a brand or one or more keywords that can be associated withthe brand. In some implementations, the third-party content item can bean image ad that includes an image that includes content that can beidentified with a particular brand.

In some implementations, the content selection module 130 can runmultiple brand uplift experiments at any given time and therefore, mayneed to check if any of the third-party content items identified by thecontent selection module 130 are associated with a brand participatingin brand uplift experiment. In some implementations, the contentselection module 130 can identify, for each third-party content itemidentified by the content selection module that is suitable for displaywith the search results, a brand associated with the third-party contentitem and determine if the brand is participating in the brand upliftexperiment.

In some implementations, the brand uplift experiment can have two arms,an experimental arm and a control arm. The experimental arm can includea first set of device identifiers, while the control arm can include asecond set of device identifiers different from the first set ofidentifiers. The brand uplift experiment can be specific to a particularbrand or advertiser. Each brand uplift experiment can correspond to alist of brand keywords associated with the brand for which the branduplift experiment is being performed. As described above, the list ofbrand keywords can be generated by the data processing system 110 orreceived from another entity, such as the advertiser. A brand keywordcan be a keyword that identifies the brand associated with the branduplift experiment. In some implementations, the brand keyword can be atrademark, keyword, phrase, acronym, or set of characters or words thatthe brand or advertiser associates with the brand itself. In someimplementations, the data processing system 110 can generate a list ofbrand keywords for the advertiser of the brand uplift experiment bycrawling one or more websites of the advertiser or brand associated withthe brand uplift experiment.

In response to determining that a third-party content item identified bythe content selection module 130 is associated with a brandparticipating in a brand uplift experiment, the content selection module130 can determine whether or not the device identifier associated withthe search query used to identify the third-party content item iseligible to participate in the brand uplift experiment. To do so, thecontent selection module 130 can determine if the device identifier isdeemed to already be aware of the brand associated with the brand upliftexperiment. In some implementations, to determine if the deviceidentifier is already aware of the brand, the content selection module130 can perform a lookup of the device identifier's previous searchqueries from a search query log to see if any of the search queriesmatch a brand keyword included in the list of brand keywords associatedwith the brand uplift experiment. The content selection module 130 maydetermine if the device identifier has previously made search queriesthat include one or more brand keywords included in the list of brandkeywords associated with the brand uplift experiment. If the deviceidentifier has previously made search queries that include the brandkeywords, there may be a presumption that the device identifier wasalready aware of the brand and therefore, may bias the experiment. Todetermine if the device identifier has previously made search queriesthat include one or more brand keywords included in the list of brandkeywords associated with the brand uplift experiment, the contentselection module 130 can request to receive a search query logcorresponding to the device identifier. The search query log can includeeach search query requested by the computing device associated with thedevice identifier. In some implementations, the search query log can belimited to search queries submitted within a predetermined time periodbefore the request for third-party content being handled by the contentselection module 130. The content selection module 130 can, for arequest for third-party content, identify the device identifier,retrieve the search query log of the device identifier, and determine ifthe device identifier made any search queries that included a keywordincluded in the list of brand keywords associated with the brand upliftexperiment.

In some implementations, the content selection module 130 can employother techniques to determine if the device identifier is already awareof the brand. If the content selection module 130 determines that thedevice identifier is already aware of the brand, the device identifiercannot participate in the brand uplift experiment for this particularbrand. As such, the content selection module 130 can provide thethird-party content item for display to the end user computing deviceassociated with the device identifier if the third-party content item isselected for display.

As described above, in some implementations, the brand uplift experimentcan have one or more parameters according to which the brand upliftexperiment is performed. In some implementations, the advertiser may setgeographic parameters, time of day parameters, budgetary parameters,among others. For instance, the advertiser may choose to perform thebrand uplift experiment on end user computing devices identified asbeing in Massachusetts. The advertiser may choose to perform the branduplift experiment during the hours of 7 pm to 11 pm on Tuesdays,perhaps, because that is when a TV show for which the brand upliftexperiment is being performed is aired on TV. In addition, theadvertiser may otherwise set parameters, for example, to limit theexperiment to a predetermined threshold, for example, 20%, of all searchqueries in which ads of the advertiser may be displayed. Otherparameters may also be used to limit the pool of device identifiers thatare eligible to participate in the brand uplift experiment.

Responsive to the content selection module 130 determining that thedevice identifier can participate in the brand uplift experiment, thecontent selection module 130 may determine if the device identifier isto be included in an experimental arm or the control arm of the branduplift experiment. In some implementations, the content selection module130 can randomly assign a device identifier to the experimental arm orcontrol arm. In some implementations, the content selection module 130can alternately assign device identifiers to the experimental arm or thecontrol arm in the order in which the requests for content is receivedby the content selection module 130.

The content selection module 130 can be configured to provide athird-party content item identified as promoting the brand correspondingto the brand uplift experiment upon assigning, to the experimental arm,the device identifier of the end user computing device to which thethird-party content item is being provided. In this way, each deviceidentifier included in the experimental arm of the brand upliftexperiment can be exposed to third-party content items that promote aparticular brand. In some implementations, the third-party content itemscan include content that includes one or more brand keywords that thebrand uplift experiment will use to measure brand uplift.

Conversely, the content selection module 130 may not provide third-partycontent items of the brand for which the brand uplift experiment isbeing performed to end user computing devices associated with deviceidentifiers included in the control arm. In some implementations, thecontrol arm and the experimental arm can be reversed. In suchimplementations, the content selection module 130 may providethird-party content items of the brand for which the brand upliftexperiment is being performed to end user computing device associatedwith device identifiers included in the control arm and not providethird-party content items of the brand for which the brand upliftexperiment is being performed to end user computing device associatedwith device identifiers included in the experimental arm.

The content selection module 130 can be configured to determine todisplay third-party content items of the first set of content items to afirst subset of the plurality of computing devices. The first set ofcontent items can include third-party content items that are suitablefor display with the search results to the search query. Each of thefirst set of content items can identify the brand corresponding to thebrand uplift experiment. The first set of the plurality of computingdevices can correspond to end user computing devices that are associatedwith device identifiers that are assigned to the experimental arm. Asstated above, the content selection module 130 can assign a deviceidentifier to one of the experimental arm or the control arm. In someimplementations, the content selection module 130 can assign the deviceidentifier responsive to determining that the device identifier iseligible to participate in the experiment. If the device identifier isassigned to the experimental arm, the content selection module 130 candetermine that the third-party content item associated with the brandcan be provided for display at the end user computing device associatedwith the device identifier.

The content selection module 130 can also be configured to determine tonot display the first set of content items to a remaining subset of theplurality of computing devices that are associated with a remainingsubset of the device identifiers. The remaining subset of the deviceidentifiers can be assigned to the control arm. The content selectionmodule 130 can be configured to select, for the remaining subset of theplurality of computing devices, third-party content items of a secondset of content items. None of the second set of content items mayidentify or promote the brand associated with the brand upliftexperiment.

For each request for content received by the data processing system 110,the content selection module 130 can be configured to provide fordisplay, third-party content items at end user computing devices. Basedon whether the device identifier associated with a given end usercomputing device is assigned to the control arm or the experimental arm,the content selection module 130 can provide third-party content itemscorresponding to the brand corresponding to the brand uplift experiment.If the device identifier is assigned to the experimental arm, thecontent selection module 130 proceeds to send the third-party contentitems that promote the brand for display. Conversely, if the deviceidentifier is assigned to the control arm, the content selection module130 can identify third-party content items promoting the brand andprevent the third-party content item from displaying. In someimplementations, the content selection module can identify a replacementcontent item that does not promote the brand and provide the replacementcontent item for display.

The brand uplift calculation module 135 can be configured to identify,from a search query log including search queries associated with thedevice identifiers of the end user computing devices that are assignedto either the experimental arm or the control arm, a portion of thefirst subset of device identifiers that performed a search using asearch query that included a keyword from a predetermined set of brandkeywords associated with the brand. To do so, the brand upliftcalculation module 135 can identify the search query logs of each of thedevice identifiers included in the experimental arm and the control arm.The brand uplift calculation module 135 can then extract datacorresponding to search queries made after the third-party content itemcorresponding to the brand for which the brand uplift experiment isperformed is either provided for display to device identifiers of theexperimental arm or not provided for display for device identifiers ofthe control arm. In some implementations, the data corresponding to thesearch queries can be limited to a predetermined time period from thetime the content selection module 130 either displayed the third-partycontent item or determined not to display the third-party content item.

In some implementations, the content selection module 130 can maintain alog of each third-party content item provided for display. The log canidentify one or more of the third-party content item, the brand which itpromotes, the time and date at which the third-party content item wasprovided, the device identifier of the end user computing device towhich the third-party content item was provided, the search query usedto identify the third-party content item and an indication if thethird-party content item is a replacement content item for anotherthird-party content item that the content selection module 130determined to not provide for display because the device identifier wasassigned to the control group, among others.

In some implementations, the content selection module 130 can beconfigured to determine the performance of each of the third-partycontent items provided for display. In some implementations, the contentselection module 130 may be configured to determine a position withinthe information resource in which the third-party content item wasdisplayed. In some implementations, the content selection module 130 maybe configured to determine a length of time the third-party content itemwas displayed on the information resource before an action to navigateto another information resource was taken on the end user computingdevice. In this way, the content selection module 130 can determine if auser at the end user computing device had a chance to glance or see thethird-party content item provided for display. If, for instance, theuser takes an action that navigates the browser to a differentinformation resource within a predetermined time period, such as 3seconds, the content selection module 130 can determine that the contentitem provided for display did not receive a valid impression.Accordingly, for the purposes of the brand uplift experiment, thecontent selection module should not include the device identifier ofthis end user computing device in the experimental arm as the userlikely did not have a chance to see and process the third-party contentitem. Similarly, if the third-party content item is of low quality interms of relevance to the search term, the content selection module 130can also not include the device identifier associated with the lowquality third-party content item in the experimental arm for thepurposes of determining the first brand awareness metric value. Otherrules may be defined that may allow the content selection module 130 toignore a device identifier in the calculation of the first brandawareness metric value if the content item was unable to get the user'sattention. In some implementations, the content selection module 130 canupdate the log of each third-party content item provided for display toreflect third-party content item impressions that may be deemed to beinvalid.

In some implementations, the brand uplift calculation module 135 canretrieve the search query logs of each of the device identifiersincluded in the brand uplift experiment on a periodic basis, forexample, daily, weekly, among others. The brand uplift experiment can bescheduled for a particular length of time and at the end of the branduplift experiment, the brand uplift calculation module 135 can beginprocessing data to determine one or more metrics related to branduplift. In some implementations, the search query log can include searchqueries associated with a device identifier that were submitted with asearch request after the third-party content item promoting the brandwas displayed at the end user computing device associated with thedevice identifier.

The brand uplift calculation module 135 can be configured to identify,for a given device identifier of the experimental arm, from a searchquery log corresponding to the device identifier, whether or not thedevice identifier performed a search query that included a keyword inthe list of brand keywords. The brand uplift calculation module 135 cananalyze entries in the search query log from the time the contentselection module 130 provided the third-party content item promoting thebrand for display till the end of the experiment.

The brand uplift calculation module 135 can be configured to determine afirst brand awareness metric value by determining a number of deviceidentifiers in the experimental arm that performed a search query thatincluded a keyword included in the list of brand keywords. As such, thebrand uplift calculation module 135 can identify and analyze, for eachof the device identifiers assigned to the experimental arm, the searchquery logs of the device identifier to identify if the search query logincludes a search query that included a keyword from the predeterminedset of brand keywords associated with the brand after the contentselection module 130 provided the third-party content item promoting thebrand for display at the end user computing device associated with thedevice identifier. The brand uplift calculation module 135 can thendetermine a ratio of the number of device identifiers that performed asearch query that included a keyword from the predetermined set of brandkeywords associated with the brand to a total number of deviceidentifiers included in the experimental arm of the brand upliftexperiment of the brand. In some implementations, the ratio can quantifybrand recall for the experimental arm.

In some implementations, the data processing system 110 can determinethe first brand awareness metric value by providing, to the deviceidentifiers assigned to the experimental arm, a survey includinginformation associated with the brand and receiving, from the deviceidentifiers of the experimental arm, survey responses. The dataprocessing system can then determine, from the survey responses, a ratioof the number of device identifiers assigned to the experimental armthat identified the brand to the total number of device identifiersassigned to the experimental arm.

The brand uplift calculation module 135 can be configured to identify,for a given device identifier of the control arm, from a search querylog corresponding to the device identifier, whether or not the deviceidentifier performed a search query that included a keyword in the listof brand keywords. The brand uplift calculation module 135 can analyzeentries in the search query log from the time the content selectionmodule 130 determined not provide a third-party content item promotingthe brand for display till the end of the experiment. In someimplementations, the time at which the content selection module 130determined to not provide a third-party content item promoting the brandfor display can be a roughly the same as a time at which the contentselection module 130 provided for display, a replacement third-partycontent item not promoting the brand.

The brand uplift calculation module 135 can be configured to determine asecond brand awareness metric value by determining a number of deviceidentifiers in the control arm that performed a search query thatincluded a keyword included in the list of brand keywords. As such, thebrand uplift calculation module 135 can identify and analyze, for eachof the device identifiers assigned to the control arm, the search querylogs of the device identifier to identify if the search query logincludes a search query that included a keyword from the predeterminedset of brand keywords associated with the brand after the contentselection module 130 determined to not provide the third-party contentitem promoting the brand for display at the end user computing deviceassociated with the device identifier. The brand uplift calculationmodule 135 can then determine a ratio of the number of deviceidentifiers that performed a search query that included a keyword fromthe predetermined set of brand keywords associated with the brand to atotal number of device identifiers included in the control arm of thebrand uplift experiment of the brand. In some implementations, the ratiocan quantify brand recall for the control arm.

In some implementations, the data processing system 110 can determinethe second brand awareness metric value by providing, to the deviceidentifiers assigned to the control arm, a survey including informationassociated with the brand and receiving, from the device identifiers ofthe control arm, survey responses. The data processing system can thendetermine, from the survey responses, a ratio of the number of deviceidentifiers assigned to the control arm that identified the brand to thetotal number of device identifiers assigned to the control arm.

The brand uplift calculation module 135 can be configured to compute abrand uplift metric value indicating an uplift in brand awareness of thebrand based on the first brand awareness metric value and the secondbrand awareness metric value. The brand uplift metric value can be apercentage change between the first brand awareness metric value and thesecond brand awareness metric value. In some implementations, to computethe brand uplift metric value, the brand uplift calculation module 135can determine the difference in the first brand awareness metric valueand the second brand awareness metric value, determine a ratio of thedifference to the second brand awareness metric value and multiple theratio by 100 to get a percentage change in the brand awareness metricvalue of the experimental arm to the control arm.

The brand uplift calculation module 135 can be configured to display, toa content provider of the first set of content items, the brand upliftmetric value. In some implementations, the brand uplift calculationmodule 135 can provide the brand uplift metric value in an informationresource through which the content provider can manage one or moreadvertising campaigns.

Referring again to FIG. 2A, the information resource 202 shown in FIG.2A includes three content items 212, 214 and 216. In one instance, theinformation resource 202 is provided for display to an end usercomputing device associated with a device identifier assigned to anexperimental arm of a brand uplift experiment for Advertiser2. As such,the content item 214 is inserted in the content slot 213 and identifiesthe brand corresponding to Advertiser2. In FIG. 2A, the keyword ‘SuperFast Money Transfer’ can be a brand keyword of the advertiser,Advertiser2. A similar content item 222 is shown in the region 220 thatincludes the brand keyword ‘Super Fast Money Transfer.’

Referring now to FIG. 2B, FIG. 2B is a display 250 depicting an onlinesearch result document 252 including a third-party content item notidentifying the first brand. The search result document 252 is similarto the document 202 but differs in that the search result document 252is provided for display to an end user computing device associated witha device identifier assigned to an control arm of the brand upliftexperiment for Advertiser2. As such, for the same search query, theinformation resource does not include the third-party content items 214or 222 corresponding to Advertiser2. Rather, replacement third-partycontent items 264 and 266 are provided for display instead of thecontent items 214 and 224 shown in FIG. 2A.

In this brand uplift experiment, the data processing system 110 candetermine whether a search query log corresponding to the deviceidentifier assigned to the experimental arm includes a search query‘Super Fast Money Transfer,’ to see if the device identifier assigned tothe experimental arm recalled the brand keyword included in thethird-party content item 214. To rule out the possibility that the userin the experimental arm identified the brand keyword ‘Super Fast MoneyTransfer’ from any other content item provided on the informationresource, the data processing system 110 determines identifier whether asearch query log corresponding to the device identifier assigned to thecontrol arm includes a search query ‘Super Fast Money Transfer.’ If theuser of the experimental arm performs a search query with the brandkeyword and the user of the control arm does not, the data processingsystem 110 can assume that the user of the experimental arm learned ofthe brand keyword from the third-party content items 214 or 222 and canuse this to calculate a brand uplift metric value of third-party contentitems of Advertiser2 that included the keyword ‘Super Fast MoneyTransfer.’

FIGS. 3A and 3B are flow diagrams depicting a method of measuring branduplift for search campaigns in a computer network environment. Themethod 300 can receive a plurality of requests for content from aplurality of computing devices associated with device identifiers (BLOCK305). The method 300 can identify, for each of the plurality of requestsfor content, a third-party content item of a first set of content itemssuitable for display with the content associated with the request (BLOCK310). The method 300 can determine that a first brand promoted by theidentified third-party content item is participating in a brand upliftexperiment (BLOCK 315). The method 300 can determine, responsive toreceiving the requests for content, to display third-party content itemsof the first set of content items to a first subset of the plurality ofcomputing devices, the first subset of the plurality of computingdevices associated with a first subset of the device identifiers (BLOCK320). The method 300 can provide, for display, the third-party contentitems of the first set of content items at the first subset of theplurality of computing devices responsive to determining to display thethird-party content items of the first set of content items to the firstsubset of the plurality of computing devices (BLOCK 325). The method 300can determine to not display the first set of content items to aremaining subset of the plurality of computing devices that areassociated with a remaining subset of the device identifiers (BLOCK330). The method 300 can select, for the remaining subset of theplurality of computing devices, third-party content items of a secondset of content items, none of the second set of content itemsidentifying the first brand (BLOCK 335). The method 300 can identify,from a search query log including search queries associated with thedevice identifiers corresponding to the plurality of computing devices,a portion of the first subset of device identifiers that performed asearch using a search query that included a keyword from a predeterminedset of brand keywords associated with the first brand (BLOCK 340). Themethod 300 can determine a first brand awareness metric value based on aratio of the portion of the first subset of device identifiers to thefirst subset of device identifiers (BLOCK 345). The method 300 canidentify, from the search query log, a portion of the second subset ofdevice identifiers that performed a search using a search query thatincluded a keyword from the predetermined set of brand keywordsassociated with the first brand (BLOCK 350). The method 300 candetermine a second brand awareness metric value based on a ratio of theportion of the second subset of device identifiers to the second subsetof device identifiers (BLOCK 355). The method 300 can compute a branduplift metric value indicating an uplift in brand awareness of the firstbrand identified by the first set of content items based on the firstbrand awareness metric value and the second brand awareness metric value(BLOCK 360). The method 300 can display, to a content provider of thefirst set of content items, the brand uplift metric value (BLOCK 365).

The data processing system can receive a plurality of requests forcontent from a plurality of computing devices associated with deviceidentifiers (BLOCK 305). Each request corresponds to a request toprovide third-party content items for display at an end user computingdevice. Each request can identify a search query based on which the dataprocessing system can select content items. The request also identifiesa device identifier, which the data processing system can use todetermine whether or not to provide one or more of the content itemsselected by the data processing system.

The data processing system can receive a request for content in responseto a request from an end user computing device. The request for contentcan include a request to provide third-party content alongside searchresults that are to be displayed in a search results informationresource. In some implementations, the requests for content can bereceived directly from end user computing devices associated with deviceidentifiers. In some implementations, the requests for content can bereceived via another entity, such as the content publisher responsive tothe content publisher receiving a request for search results based on asearch query. In some implementations, the end user computing device cantransmit the request for search results based on the search query to thecontent publisher via the network. In turn, the content publisher cantransmit a request to the data processing system for third-party contentitems to provide in the search results information resource includingthe search results generated by the content publisher responsive to thesearch query. The data processing system can then select one or morethird-party content items provided by third-party content providers fordisplay on the search results information resource.

In some implementations, the request for search results generated by theend user computing device can include the search query. In addition, therequest for search results can include a device identifier identifyingthe end user computing device. In some implementations, the deviceidentifier can be unique to the end user computing device. In someimplementations, the device identifier can be unique to a user of theend user computing device. In some implementations, the deviceidentifier can be unique to a particular application, for example, a webbrowser of the end user computing device. The request for third-partycontent received by the data processing system can include the deviceidentifier associated with the end user computing device that requestedsearch results. In some implementations, the end user computing devicecan transmit the request for third-party content to the data processingsystem. In some implementations, the content publisher receiving therequest for search results from the end user computing device cantransmit the request for third-party content to the data processingsystem.

The data processing system can identify, for each of the plurality ofrequests for content, a third-party content item of a first set ofcontent items suitable for display with the content associated with therequest (BLOCK 310). In some implementations, the data processing systemcan identify the search query included in the request and the deviceidentifier identified in the request responsive to receiving a givenrequest of the plurality of requests for third-party content. The dataprocessing system can then identify, via an advertising auction system,one or more third-party content items that are suitable for display withthe search results. In some implementations, the data processing systemcan identify third-party content items via an auction. In someimplementations, an auction can be based on the search query, such thatadvertisers can provide bids to serve third-party content items on thesame information resource as search results to the search query providedby the content publisher. The highest bidding content items may then beselected for display.

In some implementations, the data processing system can identify, forthe given request, at least one third-party content item of a first setof content items suitable for display on the information resourceincluding the search results to the search query. Each of the first setof content items can identify a brand participating in a brand upliftexperiment. In some implementations, each of the first set of contentitems can promote the same brand participating in the brand upliftexperiment. For a given third-party content item of the third-partycontent items of the first set of content items identified by the dataprocessing system as suitable for display with search resultscorresponding to the search query, the data processing system candetermine the brand the third-party content item is promoting. The dataprocessing system can do so by analyzing the content of the third-partycontent item or by analyzing the web address or contents of the landingpage to which the third-party content item is linked. In someimplementations, the data processing system can determine the brand oradvertiser the third-party content item is promoting by identifying anadvertisement campaign in which the third-party content item isincluded.

The data processing system can determine that the brand corresponding tothe third-party content item is participating in a brand upliftexperiment. Once the data processing system determines the identity ofthe brand or advertiser associated with the content item as describedabove, the data processing system can determine if the brand isparticipating in a brand uplift experiment. In some implementations, thedata processing system can perform a lookup in a database or table thatincludes a list of brands that are participating in a brand upliftexperiment. If the brand is included in the list of brands, the dataprocessing system can determine that the brand is participating in abrand uplift experiment.

The data processing system can determine that a first brand promoted bythe identified third-party content item is participating in a branduplift experiment (BLOCK 315). To do so, once the data processing systemdetermines the identity of the brand or advertiser associated with thecontent item as described above, the data processing system candetermine if the brand is participating in a brand uplift experiment byperforming a lookup in a database or table that includes a list ofbrands that are participating in a brand uplift experiment. If the brandis included in the list of brands, the data processing system candetermine that the brand is participating in a brand uplift experiment.

If the third-party content item is associated with a brand on the listof brands participating in a brand uplift experiment, the dataprocessing system has to execute a set of instructions corresponding tothe brand uplift experiment. In some implementations, the dataprocessing system can maintain a list of brands that are participatingin a brand uplift experiment. It should be appreciated that thethird-party content items can include content that identifies aparticular brand. In some implementations, the third-party content itemcan include text that identifies a brand or one or more keywords thatcan be associated with the brand. In some implementations, thethird-party content item can be an image ad that includes an image thatincludes content that can be identified with a particular brand.

The data processing system can determine, responsive to receiving therequests for content, to display third-party content items of the firstset of content items to a first subset of the plurality of computingdevices, the first subset of the plurality of computing devicesassociated with a first subset of the device identifiers (BLOCK 320).The first set of content items can include third-party content itemsthat are suitable for display with the search results to the searchquery. Each of the first set of content items can identify the brandcorresponding to the brand uplift experiment. The first set of theplurality of computing devices can correspond to end user computingdevices that are associated with device identifiers that are assigned tothe experimental arm. As stated above, the content selection module 130can assign a device identifier to one of the experimental arm or thecontrol arm. In some implementations, the content selection module 130can assign the device identifier responsive to determining that thedevice identifier is eligible to participate in the experiment. If thedevice identifier is assigned to the experimental arm, the contentselection module 130 can determine that the third-party content itemassociated with the brand can be provided for display at the end usercomputing device associated with the device identifier.

The data processing system can determine to not display the first set ofcontent items to a remaining subset of the plurality of computingdevices that are associated with a remaining subset of the deviceidentifiers (BLOCK 330). The remaining subset of the device identifierscan be assigned to the control arm. The data processing system canselect, for the remaining subset of the plurality of computing devices,third-party content items of a second set of content items. None of thesecond set of content items may identify or promote the brand associatedwith the brand uplift experiment. In some implementations, the dataprocessing system can select a content item with the next highestrelevancy score

The data processing system can select, for the remaining subset of theplurality of computing devices, third-party content items of a secondset of content items (BLOCK 335). None of the second set of contentitems may identify or promote the first brand. For each request forcontent received by the data processing system, the data processingsystem can provide for display, third-party content items at end usercomputing devices. Based on whether the device identifier associatedwith a given end user computing device is assigned to the control arm orthe experimental arm, the data processing system can provide third-partycontent items corresponding to the brand corresponding to the branduplift experiment. If the device identifier is assigned to theexperimental arm, the data processing system proceeds to send thethird-party content items that promote the brand for display.Conversely, if the device identifier is assigned to the control arm, thedata processing system can prevent the third-party content itempromoting the brand from displaying. In some implementations, the dataprocessing system can identify a replacement content item that does notpromote the brand and provide the replacement content item for display.

The data processing system can identify, from a search query logincluding search queries associated with the device identifierscorresponding to the plurality of computing devices, a portion of thefirst subset of device identifiers that performed a search using asearch query that included a keyword from a predetermined set of brandkeywords associated with the first brand (BLOCK 340). In someimplementations, the first subset of device identifiers correspond tothe device identifiers assigned to the experimental arm. The dataprocessing system can identify the search query logs of each of thedevice identifiers included in the experimental arm and the control arm.The data processing system can then extract data corresponding to searchqueries made after the third-party content item corresponding to thebrand for which the brand uplift experiment is performed is eitherprovided for display to device identifiers of the experimental arm ornot provided for display for device identifiers of the control arm. Insome implementations, the data corresponding to the search queries canbe limited to a predetermined time period from the time the contentselection module 130 either displayed the third-party content item ordetermined not to display the third-party content item. In someimplementations, the search query log includes search queries associatedwith a device identifier that were submitted with a search request afterthe third-party content item of the first set of content items wasdisplayed at a computing device associated with the device identifier.

In some implementations, the data processing system can maintain a logof each third-party content item provided for display. The log canidentify one or more of the third-party content item, the brand which itpromotes, the time and date at which the third-party content item wasprovided, the device identifier of the end user computing device towhich the third-party content item was provided, the search query usedto identify the third-party content item and an indication if thethird-party content item is a replacement content item for anotherthird-party content item that the data processing system determined tonot provide for display because the device identifier was assigned tothe control group, among others.

In some implementations, the data processing system can determine theperformance of each of the third-party content items provided fordisplay. In some implementations, the data processing system candetermine a position within the information resource in which thethird-party content item was displayed. In some implementations, thedata processing system can determine a length of time the third-partycontent item was displayed on the information resource before an actionto navigate to another information resource was taken on the end usercomputing device. In this way, the data processing system can determineif a user at the end user computing device had a chance to glance or seethe third-party content item provided for display. If, for instance, theuser takes an action that navigates the browser to a differentinformation resource within a predetermined time period, such as 3seconds, the data processing system can determine that the content itemprovided for display did not receive a valid impression. Accordingly,for the purposes of the brand uplift experiment, the content selectionmodule should not include the device identifier of this end usercomputing device in the experimental arm as the user likely did not havea chance to see and process the third-party content item. Similarly, ifthe third-party content item is of low quality in terms of relevance tothe search term, the data processing system can also not include thedevice identifier associated with the low quality third-party contentitem in the experimental arm for the purposes of determining the firstbrand awareness metric value. Other rules may be defined that may allowthe data processing system to ignore a device identifier in thecalculation of the first brand awareness metric value if the contentitem was unable to get the user's attention. In some implementations,the data processing system can update the log of each third-partycontent item provided for display to reflect third-party content itemimpressions that may be deemed to be invalid.

In some implementations, the data processing system can retrieve thesearch query logs of each of the device identifiers included in thebrand uplift experiment on a periodic basis, for example, daily, weekly,among others. The brand uplift experiment can be scheduled for aparticular length of time and at the end of the brand uplift experiment,the data processing system can begin processing data to determine one ormore metrics related to brand uplift. In some implementations, thesearch query log can include search queries associated with a deviceidentifier that were submitted with a search request after thethird-party content item promoting the brand was displayed at the enduser computing device associated with the device identifier.

The data processing system can be configured to identify, for a givendevice identifier of the experimental arm, from a search query logcorresponding to the device identifier, whether or not the deviceidentifier performed a search query that included a keyword in the listof brand keywords. The data processing system can analyze entries in thesearch query log from the time the data processing system provided thethird-party content item promoting the brand for display till the end ofthe experiment.

The data processing system can determine a first brand awareness metricvalue based on a ratio of the portion of the first subset of deviceidentifiers to the first subset of device identifiers (BLOCK 345). Thedata processing system can determine a first brand awareness metricvalue by determining a number of device identifiers in the experimentalarm that performed a search query that included a keyword included inthe list of brand keywords. As such, the data processing system canidentify and analyze, for each of the device identifiers assigned to theexperimental arm, the search query logs of the device identifier toidentify if the search query log includes a search query that included akeyword from the predetermined set of brand keywords associated with thebrand after the data processing system provided the third-party contentitem promoting the brand for display at the end user computing deviceassociated with the device identifier. The data processing system canthen determine a ratio of the number of device identifiers thatperformed a search query that included a keyword from the predeterminedset of brand keywords associated with the brand to a total number ofdevice identifiers included in the experimental arm of the brand upliftexperiment of the brand. In some implementations, the ratio can quantifybrand recall for the experimental arm

In some implementations, the data processing system can maintain, forthe device identifiers assigned to the experimental arm, a counteridentifying a number of computing devices associated with the deviceidentifiers assigned to the experimental arm on which any of thethird-party content items of the first set of content items wasdisplayed. The data processing system can increment the counter when athird-party content item that is identified by the data processingsystem to provide for display and includes content promoting the brandreceives a valid impression on a computing device on which none of thefirst set of content items has previously received a valid impression.In some implementations, the data processing system can increment thecounter when a third-party content item that is identified by the dataprocessing system to provide for display and includes content promotingthe brand receives a valid impression on a computing device on whichnone of the first set of content items has previously received a validimpression during the course of the brand uplift experiment. In somesuch implementations, to increment the counter when a content itemreceives a valid impression, the data processing system can determine alength of time from a time a third-party content item of the first setof content items was displayed on an information resource and a requestto access another content item included in the information resource isreceived and determine that the length of time is above a thresholdlength of time. If the length of time is below the threshold length oftime, the data processing system may deem the impression to be invalidfor the purposes of the brand uplift experiment.

In some implementations, the data processing system can determine thefirst brand awareness metric value by providing, to the deviceidentifiers assigned to the experimental arm, a survey includinginformation associated with the brand and receiving, from the deviceidentifiers of the experimental arm, survey responses. The dataprocessing system can then determine, from the survey responses, a ratioof the number of device identifiers assigned to the experimental armthat identified the brand to the total number of device identifiersassigned to the experimental arm.

The data processing system can identify, from the search query log, aportion of the second subset of device identifiers that performed asearch using a search query that included a keyword from thepredetermined set of brand keywords associated with the first brand(BLOCK 350). The data processing system can identify, for a given deviceidentifier of the control arm, from a search query log corresponding tothe device identifier, whether or not the device identifier performed asearch query that included a keyword in the list of brand keywords. Thedata processing system can analyze entries in the search query log fromthe time the data processing system can determined to not provide athird-party content item promoting the brand for display till the end ofthe experiment. In some implementations, the time at which the dataprocessing system determined to not provide a third-party content itempromoting the brand for display can be a roughly the same as a time atwhich the data processing system provided for display, a replacementthird-party content item not promoting the brand.

The data processing system can determine a second brand awareness metricvalue based on a ratio of the portion of the second subset of deviceidentifiers to the second subset of device identifiers (BLOCK 355). Thedata processing system can determine a second brand awareness metricvalue by determining a number of device identifiers in the control armthat performed a search query that included a keyword included in thelist of brand keywords. As such, the data processing system can identifyand analyze, for each of the device identifiers assigned to the controlarm, the search query logs of the device identifier to identify if thesearch query log includes a search query that included a keyword fromthe predetermined set of brand keywords associated with the brand afterthe data processing system determined to not provide the third-partycontent item promoting the brand for display at the end user computingdevice associated with the device identifier. The data processing systemcan then determine a ratio of the number of device identifiers thatperformed a search query that included a keyword from the predeterminedset of brand keywords associated with the brand to a total number ofdevice identifiers included in the control arm of the brand upliftexperiment of the brand. In some implementations, the ratio can quantifybrand recall for the control arm.

In some implementations, the data processing system can determine thesecond brand awareness metric value by providing, to the deviceidentifiers assigned to the control arm, a survey including informationassociated with the brand and receiving, from the device identifiers ofthe control arm, survey responses. The data processing system can thendetermine, from the survey responses, a ratio of the number of deviceidentifiers assigned to the control arm that identified the brand to thetotal number of device identifiers assigned to the control arm.

The data processing system can compute a brand uplift metric valueindicating an uplift in brand awareness of the first brand identified bythe first set of content items based on the first brand awareness metricvalue and the second brand awareness metric value (BLOCK 360). The branduplift metric value can be a percentage change between the first brandawareness metric value and the second brand awareness metric value. Insome implementations, to compute the brand uplift metric value, the dataprocessing system can determine the difference in the first brandawareness metric value and the second brand awareness metric value,determine a ratio of the difference to the second brand awareness metricvalue and multiple the ratio by 100 to get a percentage change in thebrand awareness metric value of the experimental arm to the control arm.

The data processing system can display, to a content provider of thefirst set of content items, the brand uplift metric value (BLOCK 365).In some implementations, the data processing system can provide thebrand uplift metric value in an information resource through which thecontent provider can manage one or more advertising campaigns.

FIG. 4 shows the general architecture of an illustrative computer system400 that may be employed to implement any of the computer systemsdiscussed herein (including the system 110 and its components such asthe content selection module 130 and brand uplift calculation module135) in accordance with some implementations. The computer system 400can be used to provide information via the network 105 for display. Thecomputer system 400 of FIG. 4 comprises one or more processors 420communicatively coupled to memory 425, one or more communicationsinterfaces 405, and one or more output devices 410 (e.g., one or moredisplay units) and one or more input devices 415. The processors 420 canbe included in the data processing system 110 or the other components ofthe system 110 such as the content selection module 130 and the branduplift calculation module 135.

In the computer system 400 of FIG. 4, the memory 425 may comprise anycomputer-readable storage media, and may store computer instructionssuch as processor-executable instructions for implementing the variousfunctionalities described herein for respective systems, as well as anydata relating thereto, generated thereby, or received via thecommunications interface(s) or input device(s) (if present). Referringagain to the system 110 of FIG. 1, the data processing system 110 caninclude the memory 425 to store information related to the availabilityof inventory of one or more content units, reservations of one or morecontent units, among others. The memory 425 can include the database140. The processor(s) 420 shown in FIG. 4 may be used to executeinstructions stored in the memory 425 and, in so doing, also may readfrom or write to the memory various information processed and orgenerated pursuant to execution of the instructions.

The processor 420 of the computer system 400 shown in FIG. 4 also may becommunicatively coupled to or control the communications interface(s)405 to transmit or receive various information pursuant to execution ofinstructions. For example, the communications interface(s) 405 may becoupled to a wired or wireless network, bus, or other communicationmeans and may therefore allow the computer system 400 to transmitinformation to or receive information from other devices (e.g., othercomputer systems). While not shown explicitly in the system of FIG. 1,one or more communications interfaces facilitate information flowbetween the components of the system 40. In some implementations, thecommunications interface(s) may be configured (e.g., via varioushardware components or software components) to provide a website as anaccess portal to at least some aspects of the computer system 400.Examples of communications interfaces 405 include user interfaces (e.g.,web pages), through which the user can communicate with the dataprocessing system 40.

The output devices 410 of the computer system 400 shown in FIG. 4 may beprovided, for example, to allow various information to be viewed orotherwise perceived in connection with execution of the instructions.The input device(s) 415 may be provided, for example, to allow a user tomake manual adjustments, make selections, enter data, or interact in anyof a variety of manners with the processor during execution of theinstructions. Additional information relating to a general computersystem architecture that may be employed for various systems discussedherein is provided further herein.

Implementations of the subject matter and the operations described inthis specification can be implemented in digital electronic circuitry,or in computer software embodied on a tangible medium, firmware, orhardware, including the structures disclosed in this specification andtheir structural equivalents, or in combinations of one or more of them.Implementations of the subject matter described in this specificationcan be implemented as one or more computer programs, i.e., one or moremodules of computer program instructions, encoded on computer storagemedium for execution by, or to control the operation of, data processingapparatus. The program instructions can be encoded on anartificially-generated propagated signal, e.g., a machine-generatedelectrical, optical, or electromagnetic signal that is generated toencode information for transmission to suitable receiver apparatus forexecution by a data processing apparatus. A computer storage medium canbe, or be included in, a computer-readable storage device, acomputer-readable storage substrate, a random or serial access memoryarray or device, or a combination of one or more of them. Moreover,while a computer storage medium is not a propagated signal, a computerstorage medium can be a source or destination of computer programinstructions encoded in an artificially-generated propagated signal. Thecomputer storage medium can also be, or be included in, one or moreseparate physical components or media (e.g., multiple CDs, disks, orother storage devices).

The features disclosed herein may be implemented on a smart televisionmodule (or connected television module, hybrid television module, etc.),which may include a processing module configured to integrate internetconnectivity with more traditional television programming sources (e.g.,received via cable, satellite, over-the-air, or other signals). Thesmart television module may be physically incorporated into a televisionset or may include a separate device such as a set-top box, Blu-ray orother digital media player, game console, hotel television system, andother companion device. A smart television module may be configured toallow viewers to search and find videos, movies, photos and othercontent on the web, on a local cable TV channel, on a satellite TVchannel, or stored on a local hard drive. A set-top box (STB) or set-topunit (STU) may include an information appliance device that may containa tuner and connect to a television set and an external source ofsignal, turning the signal into content which is then displayed on thetelevision screen or other display device. A smart television module maybe configured to provide a home screen or top level screen includingicons for a plurality of different applications, such as a web browserand a plurality of streaming media services, a connected cable orsatellite media source, other web “channels”, etc. The smart televisionmodule may further be configured to provide an electronic programmingguide to the user. A companion application to the smart televisionmodule may be operable on a mobile computing device to provideadditional information about available programs to a user, to allow theuser to control the smart television module, etc. In alternateimplementations, the features may be implemented on a laptop computer orother personal computer, a smartphone, other mobile phone, handheldcomputer, a tablet PC, or other computing device.

The operations described in this specification can be implemented asoperations performed by a data processing apparatus on data stored onone or more computer-readable storage devices or received from othersources.

The terms “data processing apparatus”, “data processing system”, “userdevice” or “computing device” encompasses all kinds of apparatus,devices, and machines for processing data, including by way of example aprogrammable processor, a computer, a system on a chip, or multipleones, or combinations, of the foregoing. The apparatus can includespecial purpose logic circuitry, e.g., an FPGA (field programmable gatearray) or an ASIC (application-specific integrated circuit). Theapparatus can also include, in addition to hardware, code that createsan execution environment for the computer program in question, e.g.,code that constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, a cross-platform runtimeenvironment, a virtual machine, or a combination of one or more of them.The apparatus and execution environment can realize various differentcomputing model infrastructures, such as web services, distributedcomputing and grid computing infrastructures. The content selectionmodule 130 and the brand uplift calculation module 135 can include orshare one or more data processing apparatuses, computing devices, orprocessors.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, object, orother unit suitable for use in a computing environment. A computerprogram may, but need not, correspond to a file in a file system. Aprogram can be stored in a portion of a file that holds other programsor data (e.g., one or more scripts stored in a markup languagedocument), in a single file dedicated to the program in question, or inmultiple coordinated files (e.g., files that store one or more modules,sub-programs, or portions of code). A computer program can be deployedto be executed on one computer or on multiple computers that are locatedat one site or distributed across multiple sites and interconnected by acommunication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform actions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatuses can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. The essential elements of a computer area processor for performing actions in accordance with instructions andone or more memory devices for storing instructions and data. Generally,a computer will also include, or be operatively coupled to receive datafrom or transfer data to, or both, one or more mass storage devices forstoring data, e.g., magnetic, magneto-optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer canbe embedded in another device, e.g., a mobile telephone, a personaldigital assistant (PDA), a mobile audio or video player, a game console,a Global Positioning System (GPS) receiver, or a portable storage device(e.g., a universal serial bus (USB) flash drive), for example. Devicessuitable for storing computer program instructions and data include allforms of non-volatile memory, media and memory devices, including by wayof example semiconductor memory devices, e.g., EPROM, EEPROM, and flashmemory devices; magnetic disks, e.g., internal hard disks or removabledisks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Theprocessor and the memory can be supplemented by, or incorporated in,special purpose logic circuitry.

To provide for interaction with a user, implementations of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube), plasma, or LCD(liquid crystal display) monitor, for displaying information to the userand a keyboard and a pointing device, e.g., a mouse or a trackball, bywhich the user can provide input to the computer. Other kinds of devicescan be used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser.

Implementations of the subject matter described in this specificationcan be implemented in a computing system that includes a back-endcomponent, e.g., as a data server, or that includes a middlewarecomponent, e.g., an application server, or that includes a front-endcomponent, e.g., a client computer having a graphical user interface ora Web browser through which a user can interact with an implementationof the subject matter described in this specification, or anycombination of one or more such back-end, middleware, or front-endcomponents. The components of the system can be interconnected by anyform or medium of digital data communication, e.g., a communicationnetwork. Examples of communication networks include a local area network(“LAN”) and a wide area network (“WAN”), an inter-network (e.g., theInternet), and peer-to-peer networks (e.g., ad hoc peer-to-peernetworks).

The computing system such as system 400 or system 110 can includeclients and servers. For example, the data processing system 110 caninclude one or more servers in one or more data centers or server farms.A client and server are generally remote from each other and typicallyinteract through a communication network. The relationship of client andserver arises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someimplementations, a server transmits data (e.g., an HTML page) to aclient device (e.g., for purposes of displaying data to and receivinguser input from a user interacting with the client device). Datagenerated at the client device (e.g., a result of the user interaction)can be received from the client device at the server.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinventions or of what may be claimed, but rather as descriptions offeatures specific to particular implementations of the systems andmethods described herein. Certain features that are described in thisspecification in the context of separate implementations can also beimplemented in combination in a single implementation. Conversely,various features that are described in the context of a singleimplementation can also be implemented in multiple implementationsseparately or in any suitable subcombination. Moreover, althoughfeatures may be described above as acting in certain combinations andeven initially claimed as such, one or more features from a claimedcombination can in some cases be excised from the combination, and theclaimed combination may be directed to a subcombination or variation ofa subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In some cases, the actions recited in the claims can beperformed in a different order and still achieve desirable results. Inaddition, the processes depicted in the accompanying figures do notnecessarily require the particular order shown, or sequential order, toachieve desirable results.

In certain circumstances, multitasking and parallel processing may beadvantageous. Moreover, the separation of various system components inthe implementations described above should not be understood asrequiring such separation in all implementations, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products. For example, the contentselection module 130 and the brand uplift calculation module 135 can bepart of the data processing system 110, a single module, a logic devicehaving one or more processing modules, one or more servers, or part of asearch engine.

Having now described some illustrative implementations andimplementations, it is apparent that the foregoing is illustrative andnot limiting, having been presented by way of example. In particular,although many of the examples presented herein involve specificcombinations of method acts or system elements, those acts and thoseelements may be combined in other ways to accomplish the sameobjectives. Acts, elements and features discussed only in connectionwith one implementation are not intended to be excluded from a similarrole in other implementations or implementations.

The phraseology and terminology used herein is for the purpose ofdescription and should not be regarded as limiting. The use of“including” “comprising” “having” “containing” “involving”“characterized by” “characterized in that” and variations thereofherein, is meant to encompass the items listed thereafter, equivalentsthereof, and additional items, as well as alternate implementationsconsisting of the items listed thereafter exclusively. In oneimplementation, the systems and methods described herein consist of one,each combination of more than one, or all of the described elements,acts, or components.

Any references to implementations or elements or acts of the systems andmethods herein referred to in the singular may also embraceimplementations including a plurality of these elements, and anyreferences in plural to any implementation or element or act herein mayalso embrace implementations including only a single element. Referencesin the singular or plural form are not intended to limit the presentlydisclosed systems or methods, their components, acts, or elements tosingle or plural configurations. References to any act or element beingbased on any information, act or element may include implementationswhere the act or element is based at least in part on any information,act, or element.

Any implementation disclosed herein may be combined with any otherimplementation, and references to “an implementation,” “someimplementations,” “an alternate implementation,” “variousimplementation,” “one implementation” or the like are not necessarilymutually exclusive and are intended to indicate that a particularfeature, structure, or characteristic described in connection with theimplementation may be included in at least one implementation. Suchterms as used herein are not necessarily all referring to the sameimplementation. Any implementation may be combined with any otherimplementation, inclusively or exclusively, in any manner consistentwith the aspects and implementations disclosed herein.

References to “or” may be construed as inclusive so that any termsdescribed using “or” may indicate any of a single, more than one, andall of the described terms.

Where technical features in the drawings, detailed description or anyclaim are followed by reference signs, the reference signs have beenincluded for the sole purpose of increasing the intelligibility of thedrawings, detailed description, and claims. Accordingly, neither thereference signs nor their absence have any limiting effect on the scopeof any claim elements.

The systems and methods described herein may be embodied in otherspecific forms without departing from the characteristics thereof.Although the examples provided herein relate to identifying theavailability of inventory of content units, the systems and methodsdescribed herein can be applied to any program in any vertical in whicha comparison is performed. The foregoing implementations areillustrative rather than limiting of the described systems and methods.Scope of the systems and methods described herein is thus indicated bythe appended claims, rather than the foregoing description, and changesthat come within the meaning and range of equivalency of the claims areembraced therein.

What is claimed is:
 1. A method for measuring brand uplift comprising:receiving, by a data processing system including one or more processors,a plurality of requests for content from a plurality of computingdevices associated with device identifiers; identifying, for each of theplurality of requests for content, a third-party content item of a firstset of content items suitable for display with the content associatedwith the request, each of the first set of content items identifying afirst brand; determining that the first brand promoted by the identifiedthird-party content item is participating in a brand uplift experiment;determining, by the data processing system, responsive to receiving therequest for content, to display third-party content items of the firstset of content items to a first subset of the plurality of computingdevices, the first subset of the plurality of computing devicesassociated with a first subset of the device identifiers; providing, fordisplay, the third-party content items of the first set of content itemsat the first subset of the plurality of computing devices responsive todetermining to display the third-party content items of the first set ofcontent items to the first subset of the plurality of computing devices;determining to not display the first set of content items to a remainingsubset of the plurality of computing devices that are associated with aremaining subset of the device identifiers; selecting, for the remainingsubset of the plurality of computing devices, third-party content itemsof a second set of content items, none of the second set of contentitems identifying the first brand; identifying, from a search query logincluding search queries associated with the device identifierscorresponding to the plurality of computing devices, a portion of thefirst subset of device identifiers that performed a search using asearch query that included a keyword from a predetermined set of brandkeywords associated with the first brand; determining a first brandawareness metric value based on a ratio of a number of deviceidentifiers in the portion of the first subset of device identifiers toa total number of device identifiers in the first subset of deviceidentifiers; identifying, from the search query log, a portion of thesecond subset of device identifiers that performed a search using asearch query that included a keyword from the predetermined set of brandkeywords associated with the first brand; determining a second brandawareness metric value based on a ratio of a number of deviceidentifiers in the portion of the second subset of device identifiers toa total number of device identifiers in the second subset of deviceidentifiers; computing a brand uplift metric value indicating an upliftin brand awareness of the brand identified by the first set of contentitems based on the first brand awareness metric value and the secondbrand awareness metric value; and providing data based on the branduplift metric value.
 2. The method of claim 1, wherein the search querylog includes search queries associated with a device identifier thatwere submitted with a search request after the third-party content itemof the first set of content items was displayed at a computing deviceassociated with the device identifier.
 3. The method of claim 1, whereindetermining the first brand awareness metric value includes: providing,to the first subset of device identifiers, a survey includinginformation associated with the first brand; receiving, from the firstsubset of device identifiers, survey responses; and determining, fromthe survey responses, a ratio of a number of device identifiers in aportion of the first subset of device identifiers that identified thebrand to the total number of device identifiers in the first subset ofdevice identifiers, the first brand awareness metric value based on thedetermined ratio.
 4. The method of claim 1, wherein determining thesecond brand awareness metric value includes: providing, to the secondsubset of device identifiers, a survey including information associatedwith the first brand; receiving, from the second subset of deviceidentifiers, survey responses; and determining, from the surveyresponses, a ratio of a number of device identifiers in a portion of thesecond subset of device identifiers that identified the first brand tothe total number of device identifiers in the second subset of deviceidentifiers, the second brand awareness metric value based on thedetermined ratio.
 5. The method of claim 1, further comprising:maintaining, by the data processing system, for the first subset ofdevice identifiers, a counter identifying a number of computing devicesassociated with the first subset of device identifiers on which any ofthe third-party content items of the first set of content items wasdisplayed; and incrementing the counter when a content item receives avalid impression on a computing device on which none of the first set ofcontent items has previously received a valid impression.
 6. The methodof claim 5, wherein incrementing the counter when a content itemreceives a valid impression on a computing device on which none of thefirst set of content items has previously received a valid impressionincludes: determining, responsive to displaying one of the third-partycontent items of the first set of content items at the first subset ofthe plurality of computing devices, a length of time from a time athird-party content items of the first set of content items wasdisplayed on an information resource and a request to access anothercontent item included in the information resource is received; anddetermining that the length of time is above a threshold length of time.7. The method of claim 1, further comprising generating, by the dataprocessing system, the predetermined set of brand keywords associatedwith the first brand by analyzing content associated with the firstbrand.
 8. A system of measuring brand uplift in a computer networkenvironment, comprising: a data processing system having a contentselection module and a brand uplift calculation module, the dataprocessing system configured to: receive a plurality of requests forcontent from a plurality of computing devices associated with deviceidentifiers; identify, for each of the plurality of requests forcontent, a third-party content item of a first set of content itemssuitable for display with the content associated with the request, eachof the first set of content items identifying a first brand; determinethat the first brand promoted by the identified third-party content itemis participating in a brand uplift experiment; determine, responsive toreceiving the requests for content, to display third-party content itemsof the first set of content items to a first subset of the plurality ofcomputing devices, the first subset of the plurality of computingdevices associated with a first subset of the device identifiers;provide, for display, the third-party content items of the first set ofcontent items at the first subset of the plurality of computing devicesresponsive to determining to display the third-party content items ofthe first set of content items to the first subset of the plurality ofcomputing devices; determine to not display the first set of contentitems to a remaining subset of the plurality of computing devices thatare associated with a remaining subset of the device identifiers;select, for the remaining subset of the plurality of computing devices,third-party content items of a second set of content items, none of thesecond set of content items identifying the first brand; identify, froma search query log including search queries associated with the deviceidentifiers corresponding to the plurality of computing devices, aportion of the first subset of device identifiers that performed asearch using a search query that included a keyword from a predeterminedset of brand keywords associated with the first brand; determine a firstbrand awareness metric value based on a ratio of a number of deviceidentifiers in the portion of the first subset of device identifiers toa total number of device identifiers in the first subset of deviceidentifiers; identify, from the search query log, a portion of thesecond subset of device identifiers that performed a search using asearch query that included a keyword from the predetermined set of brandkeywords associated with the first brand; determine a second brandawareness metric value based on a ratio of a number of deviceidentifiers in the portion of the second subset of device identifiers toa total number of device identifiers in the second subset of deviceidentifiers; compute a brand uplift metric value indicating an uplift inbrand awareness of the first brand identified by the first set ofcontent items based on the first brand awareness metric value and thesecond brand awareness metric value; and provide data based on the branduplift metric value.
 9. The system of claim 8, wherein the search querylog includes search queries associated with a device identifier thatwere submitted with a search request after the first set of contentitems was displayed at a computing device associated with the deviceidentifier.
 10. The system of claim 8, wherein to determine the firstbrand awareness metric value, the data processing system is furtherconfigured to: provide, to the first subset of device identifiers, asurvey including information associated with the first brand; receive,from the first subset of device identifiers, survey responses; anddetermine, from the survey responses, a ratio of a number of deviceidentifiers in a portion of the first subset of device identifiers thatidentified the first brand to the total number of device identifiers inthe first subset of device identifiers, the first brand awareness metricvalue based on the determined ratio.
 11. The system of claim 8, whereinto determine the second brand awareness metric value, the dataprocessing system is further configured to: provide, to the secondsubset of device identifiers, a survey including information associatedwith the first brand; receive, from the second subset of deviceidentifiers, survey responses; and determine, from the survey responses,a ratio of a number of device identifiers in a portion of the secondsubset of device identifiers that identified the first brand to thetotal number of device identifiers in the second subset of deviceidentifiers, the second brand awareness metric value based on thedetermined ratio.
 12. The system of claim 8, wherein the data processingsystem is further configured to: maintain, for the first subset ofdevice identifiers, a counter identifying a number of computing devicesassociated with the first subset of device identifiers on which any ofthe third-party content items of the first set of content items wasdisplayed; and incrementing the counter when a content item receives avalid impression on a computing device on which none of the first set ofcontent items has previously received a valid impression.
 13. The systemof claim 12, wherein to increment the counter when a content itemreceives a valid impression on a computing device on which none of thefirst set of content items has previously received a valid impression,the data processing system is configured to: determine, responsive todisplaying one of the third-party content items of the first set ofcontent items at the first subset of the plurality of computing devices,a length of time from a time a third-party content items of the firstset of content items was displayed on an information resource and arequest to access another content item included in the informationresource is received; and determine that the length of time is above athreshold length of time.
 14. The system of claim 8, further comprisinggenerating, by the data processing system, the predetermined set ofbrand keywords associated with the first brand by analyzing contentassociated with the first brand.
 15. A computer readable storage mediumstoring instructions that when executed by one or more data processors,cause the one or more data processors to perform operations comprising:receiving, a plurality of requests for content from a plurality ofcomputing devices associated with device identifiers; identifying, foreach of the plurality of requests for content, a third-party contentitem of a first set of content items suitable for display with thecontent associated with the request, each of the first set of contentitems identifying a first brand; determining that the first brandpromoted by the identified third-party content item is participating ina brand uplift experiment; determining, responsive to receiving therequests for content, to display third-party content items of the firstset of content items to a first subset of the plurality of computingdevices, the first subset of the plurality of computing devicesassociated with a first subset of the device identifiers; providing, fordisplay, the third-party content items of the first set of content itemsat the first subset of the plurality of computing devices responsive todetermining to display the third-party content items of the first set ofcontent items to the first subset of the plurality of computing devices;determining to not display the first set of content items to a remainingsubset of the plurality of computing devices that are associated with aremaining subset of the device identifiers; selecting, for the remainingsubset of the plurality of computing devices, third-party content itemsof a second set of content items, none of the second set of contentitems identifying the first brand; identifying, from a search query logincluding search queries associated with the device identifierscorresponding to the plurality of computing devices, a portion of thefirst subset of device identifiers that performed a search using asearch query that included a keyword from a predetermined set of brandkeywords associated with the first brand; determining a first brandawareness metric value based on a ratio of a number of deviceidentifiers in the portion of the first subset of device identifiers toa total number of device identifiers in the first subset of deviceidentifiers; identifying, from the search query log, a portion of thesecond subset of device identifiers that performed a search using asearch query that included a keyword from the predetermined set of brandkeywords associated with the first brand; determining a second brandawareness metric value based on a ratio of a number of deviceidentifiers in the portion of the second subset of device identifiers toa total number of device identifiers in the second subset of deviceidentifiers; computing a brand uplift metric value indicating an upliftin brand awareness of the brand identified by the first set of contentitems based on the first brand awareness metric value and the secondbrand awareness metric value; and providing data based on the branduplift metric value.
 16. The computer readable storage medium of claim15, wherein the search query log includes search queries associated witha device identifier that were submitted with a search request after thethird-party content item of the first set of content items was displayedat a computing device associated with the device identifier.
 17. Thecomputer readable storage medium of claim 15, wherein determining thefirst brand awareness metric value includes: providing, to the firstsubset of device identifiers, a survey including information associatedwith the first brand; receiving, from the first subset of deviceidentifiers, survey responses; and determining, from the surveyresponses, a ratio of a number of device identifiers in a portion of thefirst subset of device identifiers that identified the brand to thetotal number of device identifiers in the first subset of deviceidentifiers, the first brand awareness metric value based on thedetermined ratio.
 18. The computer readable storage medium of claim 15,wherein determining the second brand awareness metric value includes:providing, to the second subset of device identifiers, a surveyincluding information associated with the first brand; receiving, fromthe second subset of device identifiers, survey responses; anddetermining, from the survey responses, a ratio of a number of deviceidentifiers in a portion of the second subset of device identifiers thatidentified the first brand to the total number of device identifiers inthe second subset of device identifiers, the second brand awarenessmetric value based on the determined ratio.
 19. The computer readablestorage medium of claim 15, further comprising: maintaining, for each ofthe first subset of device identifiers, a counter identifying a numberof times one of the third-party content items of the first set ofcontent items was displayed at the computing device associated with thedevice identifier; and incrementing the counter when a content itemreceives a valid impression on a computing device on which none of thefirst set of content items has previously received a valid impression.20. The computer readable storage medium of claim 19, further comprisinggenerating, by the data processing system, the predetermined set ofbrand keywords associated with the first brand by analyzing contentassociated with the first brand.